IN THE UNITED STATE^S'PATENT AND TRADEMARK OFFICE 




MIS 2 7 20ffJ * . 



Docket No. 500.31 754CX2 



Applicant: 



Masayuki TANI et al. 



Serial No.: 



09/845,838 



Filed: 



1 May 2001 



For: 



VIDEO OR INFORMATION PROCESSING METHOD AND 
PROCESSING APPARATUS, AND MONITORING METHOD 
AND MONITORING APPARATUS USING THE SAME 



Art Unit: 



2673 



Examiner: 



Unassigned 



PRELIMINARY AMENDMENT 



Assistant Commissioner for Patents 

Washington, D.C. 20231 27 August 2001 

Sir: 

In the matter of the above-identified application, Applicant hereby submits the 
following for entry. 

IN THE CLAIMS: 

Please cancel Claims 1-31 , 39-43 and 45-51 , without prejudice or disclaimer. 
Please amend Claims 32-35, 44 and 62 as follows. 

32. (Once Amended) An image searching method for searching a video image 
and for displaying the searched image, comprising: 

a search key designating step for designating a search key by inputting either 
a text or a pattern; and an image searching step for displaying a video image in 
which an object adapted to the search key designated by said search key 
designating step is being displayed; 
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wherein said image searching step includes a step for selecting a video 
camera based upon said search key. 

33. (Once Amended) An image searching method as claimed in Claim 32, 
wherein said image searching step includes a step for operating a video camera 
based upon the search key. 

34. (Once Amended) An image searching method as claimed in Claim 32, 
wherein said image search-step includes a step for clearly indicating an object 
adapted to the search key by synthesizing a graphics with the searched image to be 
displayed. 

35. (Once Amended) An image searching method as claimed in Claim 32, 
wherein said search key input step includes a step for inputting a search key by a 
voice. 

44. (Once Amended) An information processing apparatus for storing data to 
control an object (referred to "control data" hereinafter), and also for storing either 
audio, or video data related to said object, comprising: 

means for relating the control data with the audio, or video data; and 

means for relating the control data with the audio, or video data based upon 
said relating means to be outputted; 

wherein; 
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first reproducing means for reproducing said control data, and second 
reproducing means for reproducing at least one of said audio data and said video 
data are further comprised; 

said relating means includes means for storing information for relating said 
control data to said audio data, or said video data; 

said output means includes: first output means for outputting said reproduced 
measurement data; 

second output means for outputting said reproduced sound data, or said 
reproduced video data; and 

synchronizing means for controlling said first and second output means in 
such a manner that said outputted control data is synchronized with said audio data, 
or said video data based on said relating information to be outputted. 

62. A monitoring apparatus as claimed in Claim 44, wherein third storage 
means for storing operation input information of an operator is further comprised, 

said output means includes third output means for outputting the operation 
input information of the operator in synchronism with the reproduction of the video 
data or the audio data; 

wherein means for accepting operation input information is further comprised; 

said synchronizing means includes means for searching video and audio data 
acquired at a time instant very close to a time instant when the designated operation 
is inputted; and 
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said second output means includes means for controlling the third output 
means to output the operation input information in synchronism with the reproduction 
of the video data, or the audio data by the second output means; 

wherein said relating means includes means for storing information to relate 
the monitoring object with the control data; 

said control means includes means for selecting a monitoring object related to 
said selected control data based on said information; and 

said second output means includes means for displaying said selected 
monitoring object. 

Please add new Claims 65-102, as follows. 

65. (New) An interactive development system for developing interactive user 
interface using a video image, comprising; 

pattern drawing means for drawing a pattern on a video image displayed on a 
screen of a display means; and 

storing means for storing said pattern drawn on said video image in 
corresponding relationship with said video image. 

66. (New) A live interactive development system for developing an interactive 
user interface using a live video image from a video camera comprising; 

a pattern drawing means for drawing a pattern on a live video image 
photographed by a video camera displayed on a screen of a display means; and 

a storing means for storing said pattern drawn on said live video image in 
corresponding relationship with said live video image; 
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wherein said storing means stores said pattern drawn on said live video 
image and camera parameters used for imaging said live video image. 

67. (New) An interactive development system as claimed in Claim 66, wherein 
said storing means stores a pair of said pattern drawn on said video image and the 
identification of a camera used for imaging said video image. 

68. (New) An interactive development system as claimed in Claim 66, wherein 
said pattern drawing means allows drawing a pattern by combining basic pattern 
elements, and further comprising; 

a pattern element selecting means for selecting at least one said basic pattern 
element drawn on said video image; and 

a procedure defining means for defining a procedure for said basic pattern 
element selected by said pattern element selecting means and storing said defined 
procedure in corresponding relationship with said selected basic pattern element. 

69. (New) An interactive live video image operating system for performing a 
process with respect to a subject apparatus selected from a plurality of apparatus, a 
live real-time video image of said plurality of apparatus photographed live by a video 
camera and being displayed live on a display screen of a display means, said 
system comprising: 

a storing means for storing information about positions of said plurality of 
apparatus; 
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a designating means for designating a point or a portion of said live real-time 
video image; 

an identifying means for identifying an apparatus corresponding to said 
designated point or portion by said designating means based on said information 
stored in said means for storing; and 

a performing means for performing a process related to said identified subject 
apparatus. 

70. (New) An interactive live interactive video image operating system as 
claimed in Claim 69, wherein said information stored in said storing means is 
information representing positions of said plurality of apparatus on said live real-time 
video image or information representing positions where said plurality of apparatus 
are actually arranged, respectively. 

71. (New) An interactive live video image operating system as claimed in 
Claim 69, wherein said performing means remotely operates the identified said 
subject apparatus. 

72. (New) An interactive live video image operating system as claimed in 
Claim 69, wherein said performing means includes a process to change a video 
image based on a designated said subject apparatus. 
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73. (New) An interactive video image operating system as claimed in Claim 
69, wherein said performing means includes a process to operate a subject video 
camera based on a designated said subject apparatus. 

74. (New) An interactive video image operating system as claimed in Claim 
69, wherein said performing means includes a step of superimposing a graphics 
corresponding to a designated said subject apparatus on said video image to be 
displayed, and for providing a visual indication as to which apparatus has been 
designated. 

75. (New) An interactive video image operating system as claimed in Claim 
69, wherein said performing means includes a step of menu displaying a list of 
executable processes corresponding to a designated said subject apparatus, and a 
step of allowing selection of a process to be executed. 

76. (New) An interactive video image operating system as claimed in Claim 
69, wherein said performing means includes: 

a graphics displaying means for synthesizing a graphics corresponding to a 
designated said subject apparatus with said video image to be displayed; 

a graphics element selecting means for selecting a subject element displayed 
on said graphics; and 

an element performing means for performing a process based on said subject 
element selected by said graphics element selecting means. 
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77. (New) An interactive video image operating system as claimed in Claim 
69, wherein said designating means includes an object highlighting step for 
highlighting a designated said subject apparatus within an image. 

78. (New) An interactive video image operating system as claimed in Claim 
69, wherein said performing means includes a step of displaying information related 
to a designated said subject apparatus. 

79. (New) An interactive video image operating apparatus for performing an 
image searching process, comprising: 

a memory for storing information with respect to a plurality of apparatus;; 

a designating means for designating a key relating to a subject apparatus said 
plurality of apparatus; 

an identifying means for identifying said subject apparatus corresponding to 
said designated key based on said information stored in said memory in response to 
said designated key; and 

a photographing means for photographing an identified said subject apparatus 
by a video camera and displaying a video image of the identified said subject 
apparatus on a display screen. 

80. (New) An interactive video image operating apparatus as claimed in Claim 
79, wherein said photographing means operates said video camera based on the 
identified said subject apparatus so that the video image of the identified said subject 
apparatus is displayed on said display screen. 
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81 .(New) An interactive video image operating system as claimed in Claim 69, 
further comprising: 

a graphic image display means for displaying a graphic image on said display 
screen such that said graphic image is superimposed on said real-time video image of said 
subject apparatus wherein said storing means stores information about a position of said 
graphic image as said information about position of said apparatus, and said identifying 
means includes means for identifying a pointed graphic image with said subject apparatus 
when said graphic image is pointed by said pointing means. 

82. (New) An interactive video image operating system as claimed in Claim 69, 
wherein said performing means includes means for, when a pointed position on said 
display screen is moved on said display screen, operating said subject apparatus in a 
manner that the real-time video image of said subject apparatus traces the pointed 
position on said display screen. 

83. (New) An interactive live video image operating system as claimed in 
Claim 69, wherein said information stored in said storing means is information 
representing positions of said plurality of apparatus on said display screen or 
information representing positions where said plurality of apparatus are actually 
arranged, respectively, and that said designating means designates a position on 
said display screen of said live video image displayed live on said display screen, 
and said identifying means identifies the designated live video image with said 
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subject apparatus photographed live by said video camera based on a designated 
position on said display means and said information stored in said storing means. 

84. (New) An interactive live video image searching method as claimed in 
Claim 33, wherein said step for operating a video camera includes a substep for 
panning or tilting said video camera to photograph said subject apparatus 
corresponding to said search key designated by said search key designating step. 

85. (New) An interactive live video image searching method as claimed in 
Claim 33, wherein said step for operating a video camera includes a substep for 
zooming said video camera to photograph said subject apparatus corresponding to 
said search key designated by said search key designating step. 

86. (New) A plant monitoring apparatus, comprising: 

a storing section for storing image information of an object photographed by a 
camera which photographs a subject to be monitored and for storing process data of 
said subject to be monitored so as to correspond in time base in its contents with 
said photographed image information; 

a read section for reading from said storage section said photographed image 
information and said process data corresponding in time base in its contents with 
said photographed image information; and 

a display section for displaying said photographed image information and said 
process data corresponding in time base in its contents with said photographed 
image information read from said read section. 
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87. (New) A plant monitoring apparatus, comprising: 

a storing section for storing image information of an object photographed by a 
camera which photographs a subject to be monitored and for storing process data of 
said subject to be monitored; 

a control section for controlling, at a time of display, said photographed image 
information and said process data so as to be synchronous in its contents with said 
photographed image information; and 

a display section for displaying said photographed image information and said 
process data synchronous in its contents with said photographed image information 
read from said read section. 

88. (New) An interactive live video image operating method for operating a 
video image of at least one object displayed on a screen of display means, 
comprising: 

an object designating step for designating an object within a video image; and 
a process performing step for performing a process corresponding to the 
designated object, wherein said apparatus designating step includes an apparatus 
highlighting step for highlighting said subject apparatus designated. 

89. (New) An image operating method as claimed in Claim 88, wherein said 
object designating step includes a step for designating an object of an image 
inputted from a video camera and being displayed on said screen, and wherein said 
apparatus designating step includes an apparatus highlighting step for highlighting 
said subject apparatus designated. 
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90. (New) An interactive live video image operating system for performing a 
process with respect to a subject apparatus selected from a plurality of apparatuses, 
a live real-time video image of said plurality of apparatuses photographed live by a 
video camera being displayed live on a display screen of a display means, said 
system comprising: 

a parameter control means for controlling a parameter of said video camera; 
a storing means for storing information about positions of said plurality of 
apparatuses; 

a pointing means for pointing on said display screen to a live real-time video 
image of said subject apparatus displayed live on said display screen of said display 
means; 

an identifying means for identifying a pointed to live real-time video image with 
said subject apparatus based on said information stored in said storing means and 
said parameter of said video camera; and 

a performing means for performing a process related to an identified said 
subject apparatus, 

91. (New) An interactive live video image operating apparatus for performing 
a process with respect to a subject apparatus selected from a plurality of apparatus, 
comprising: 

a video camera for photographing at least one apparatus of said plurality of 
apparatus; 
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a display means for displaying on a display screen thereof a video image of 
said at least one apparatus, as a real time video image, which is being photographed 
by said video camera; 

a designating means for designating on said display screen the real time 
video image of said subject apparatus being displayed on said display screen of said 
display means; and 

a performing means for performing a process relating to said subject 
apparatus corresponding to the designated real time video image; 

wherein said performing means includes a step of superimposing a graphics 
corresponding to a designated said subject apparatus on said video image to be 
displayed, and for providing a visual indication as to which apparatus has been 
designated. 

92, (New) An interactive live video image operating method for performing a 
process with respect to a subject apparatus selected from a plurality of apparatus, 
comprising; 

a video camera for photographing at least one apparatus of said plurality of 
apparatus; 

a display means for displaying on a display screen thereof a video image of 
said at least one apparatus, as a real time video image, which is being photographed 
by said video camera; 

a designating means for designating on said display screen the real time 
video image of said subject apparatus being displayed on said display screen of said 
display means; and 
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a performing means for performing a process relating to said subject 
apparatus corresponding to the designated real time video image; 

wherein said designating means includes an apparatus highlighting step for 
highlighting said subject apparatus designated within an image. 

93. (New) A interactive live video image operating system as claimed in Claim 
89, wherein said parameter is at least one of an imaging direction, an attitude, an 
angle of view and a position of said video camera. 

94. (New) An interactive live video image operating system as claimed in 
Claim 69, wherein at least one of said plurality of apparatus is an instructing means 
for supplying an instruction for controlling a device to be controlled to said device, 
and said performing means operates said instructing means when said instructing 
means is designated by said designating means so that said instructing means 
supplies the instruction to said device; and 

wherein said performing means includes means for, when the live real-time 
video image of said subject apparatus is pointed to on said display screen by said 
pointing means, operating said parameter control means to change a parameter of 
said video camera. 

95. (New) An interactive live video image operating system as claimed in 
Claim 89, wherein said identifying means includes means for converting an 
information about the position of the pointed to live real-time video image into a 
converted information in accordance with the parameter of said video camera, and 
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means for identifying the pointed to live real-time video image with said subject 
apparatus based on said information stored in said storing means and the converted 
information. 

96. (New) An interactive live video image operating system as claimed in 
Claim 89, further comprising: 

a graphic image display means for displaying a graphic image on said display 
screen such that said graphic image is superimposed on said live real-time video 
image of said subject apparatus, wherein said storing means stores information 
about a position of said graphic image as said information about position of said 
apparatus, said identifying means includes means for identifying a pointed to graphic 
image with said subject apparatus when said graphic image is pointed to by said 
pointing means, and said graphic image display means includes means for, when a 
parameter of said video camera is changed, changing a display of said live real-time 
graphic image of said subject apparatus on said display screen in accordance with 
the changed parameter. 

97. (New) An interactive live video image operating system as claimed in 
Claim 89, wherein said performing means includes means for, when a pointed to 
position on said display screen is moved on said display screen, operating said 
subject apparatus in such a manner that the live real-time video image of said 
subject apparatus traces the pointed to position on said display screen. 
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98.(New) An interactive live video image operating apparatus allowing user 
designation of at least one apparatus in a live video image for initiating at least one 
operation with respect thereto, said apparatus comprising: 

at least one camera performing live photography of at least one apparatus to 
be monitored; 

a display receiving live image signals from said at least one camera, and 
displaying a live video image of said at least one apparatus; 

a data memory storing operation data for each said at least one apparatus, 
each said operation data being usable for performing said at least one operation with 
respect to a respective apparatus; 

a designating device allowing user designation of an apparatus live image on 
the displayed said live video image, for which said at least one operation is to be 
initiated; 

a relating device for determining a selected apparatus from a designated said 
apparatus live image, and determining a set of said operation data in said data 
memory related to said selected apparatus; and 

an actuating device for receiving said set of said operation data determined in 
said relating device, and using said set for performing said at least one operation 
with respect to said selected apparatus; 

and further comprising: 

a graphic image display device responsive to user designation of said 
apparatus live image, for superimposing a graphic image on said live real-time video 
image with respect to said apparatus live image so as to indicate a selected 
apparatus on said live real-time video image. 
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99. (New) An interactive live video image operating apparatus allowing user 
designation of at least one apparatus in a live video image for initiating at least one 
operation with respect thereto, said apparatus comprising: 

at least one camera performing live photography of at least one apparatus to 
be monitored; 

a display receiving live image signals from said at least one camera, and 
displaying a live video image of said at least one apparatus; 

a data memory storing operation data for each said at least one apparatus, 
each said operation data being usable for performing said at least one operation with 
respect to a respective apparatus; 

a designating device allowing user designation of an apparatus live image on 
the displayed said live video image, for which said at least one operation is to be 
initiated; 

a relating device for determining a selected apparatus from a designated said 
apparatus live image, and determining a set of said operation data in said data 
memory related to said selected apparatus; and 

an actuating device for receiving said set of said operation data determined in 
said relating device, and using said set for performing said at least one operation 
with respect to said selected apparatus; and further comprising: 

a graphic image display device responsive to user designation of said 
apparatus live image, for superimposing a graphic image on said live real-time video 
image with respect to said apparatus live image so as to provide further information 
concerning a selected apparatus on said live real-time video image. 
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100. (New) An image searching method as claimed in Claim 33, wherein said 
image search-step includes a step for clearly indicating an object adapted to the 
search key by synthesizing a graphics with the searched image to be displayed. 

101. (New) An image searching method as claimed in Claim 33, wherein said 
search key input step includes a step for inputting a search key by a voice. 

102. (New) An image searching method as claimed in Claim 34, wherein said 
search key input step includes a step for inputting a search key by a voice. 

REMARKS 

Claims 1-64 were filed with the above-identified application. Unrelated to any 
prior art rejection from any parent application , Claims 1-31, 39-43 and 45-51 have 
been cancelled without prejudice or disclaimer, Claims 32-35, 44 and 66 have been 
amended, anc | new Claims 65-102 have been added. Upon entry of this Preliminary 
Amendment, Claims 32-38, 44 and 52-102 will be pending for further consideration 
and search in the present application. 

INFORMATION DISCLOSURE STATEMENT 

Submitted concurrently herewith is an Information Disclosure Statement with 
Forms PTO-1449, for entry in the above-identified application so such information 
appears on the face of any patent issuing hereon. 
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NOTICE TO FILE MISSING PARTS 



Submitted concurrently herewith is a Response to Notice to File Missing 
Parts, along with a copy of the Notice and all of the appropriate late filing surcharge 
and claims fees authorized for payment on the attached Form PTO-2038. 

APPENDIX A-MARKED VERSION 

Attached hereto is Appendix A-Marked Version to indicate additions and 
deletions in the amended claims by underlining and bracketing, respectively. 



Applicant respectfully submits that this Preliminary Amendment is proper and 
should be entered, because it is being timely filed so as not to burden the Office with 
undue reconsideration and/or further search in connection therewith. 

No additional fees are required other than those authorized in the attached 
Form PTO-2038. However, if any Petition or fees are actually proper for the filing of 
this paper, please charge same to ATS&K Deposit Account No. 01-2135 (as Order 
No. 500.31 754CX2), and please credit any overpayment or excess fees thereto. 



CONCLUSION 



Respectfully submitted 



Paul j./fekwlerawski, Registration No. 32,173 
ANTONELLI, TERRY, STOUT & KRAUS, LLP 
1300 North Seventeenth Street, Suite 1800 
Arlington, VA 22209 
Direct Dial 703-312-6636 
Facsimile 703-312-6666 




ATTACHMENTS: 
Appendix A-Marked Version 
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APPENDIX A-MARKED VERSION 

32. (Once Amended) An image searching method [as claimed in Claim 31] for 
searching a video image and for displaying the searched image, comprising: 

a search key designating step for designating a search key by inputting either 
a text or a pattern; and an image searching step for displaying a video image in 
which an object adapted to the search key designated by said search key 
designating step is being displayed; 

wherein said image searching step includes a step for selecting a video 
camera based upon said search key. 

33. (Once Amended) An image searching method as claimed in [Claim 31 or] 
Claim 32, wherein said image searching step includes a step for operating a video 
camera based upon the search key. 

34. (Once Amended) An image searching method as claimed [any one of the 
preceding Claims 31 to 33] in Claim 32 , wherein said image search-step includes a 
step for clearly indicating an object adapted to the search key by synthesizing a 
graphics with the searched image to be displayed. 

35. (Once Amended) An image searching method as claimed [any one of the 
preceding Claims 31 to 34] in Claim 32 , wherein said search key input step includes 
a step for inputting a search key by a voice. 
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44. (Once Amended) An information processing apparatus [as claimed in 
Claim 43] for storing data to control an object (referred to "control data" hereinafter), 
and also for storing either audio, or video data related to said object, comprising: 

means for relating the control data with the audio, or video data; and 

means for relating the control data with the audio, or video data based upon 
said relating means to be outputted; 

wherein; 

first reproducing means for reproducing said control data, and second 
reproducing means for reproducing at least one of said audio data and said video 
data are further comprised; 

said relating means includes means for storing information for relating said 
control data to said audio data, or said video data; 

said output means includes: first output means for outputting said reproduced 
measurement data; 

second output means for outputting said reproduced sound data, or said 
reproduced video data; and 

synchronizing means for controlling said first and second output means in 
such a manner that said outputted control data is synchronized with said audio data, 
or said video data based on said relating information to be outputted. 

62. A monitoring apparatus as claimed in Claim [51] 44, wherein third storage 
means for storing operation input information of an operator is further comprised, 
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said output means includes third output means for outputtina the operation 
input information of the operator in synchronism with the reproduction of the video 
data or the audio data; 

wherein means for accepting operation input information is further comprised; 

said synchronizing means includes means for searching video and audio data 
acouired at a time instant very close to a time instant when the designated operation 
is inputted; and 

said second output means includes means for controlling the third output 
means to output the operation input information in synchronism with the reproduction 
of the video data, or the audio data bv the second output means; 

wherein said relating means includes means for storing information to relate 
the monitoring object with the control data; 

said control means includes means for selecting a monitoring object related to 
said selected control data based on said information; and 

said second output means includes means for displaying said selected 
monitoring object. 
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